Mechanisms utilized today to evaluate and improve the performance of search results are typically based on generating a mapping from queries to pages. For instance, one mechanism utilizes humans to measure the relevance of search results returned by a particular query. Such human relevance judgment methods, however, are mainly useful for evaluating small sets of search results and do not scale well for evaluating the performance of a search engine over a large evaluation corpora.
Another mechanism utilized to evaluate and improve the performance of search engines uses implicit measures of relevance, such as identifying clicks on search results. This mechanism is only effective, however, for pages surfaced by the search engine being evaluated. Some pages might not be surfaced by a search engine for a variety of reasons, including bad ranking, indexing problems, network issues, and others.
Consequently, if a page does not get surfaced or is not surfaced with a high enough rank, the page might never be made available for a human relevance judgment, nor will the page be made available to be clicked upon.
It is with respect to these and other considerations that the disclosure made herein is presented.